1. Field of the Invention
The present invention relates to an environment monitoring system for picking up the front, rear or diagonal-rear view from one""s own vehicle by an image pick-up means such as a video camera installed on the vehicle such as a motor car, detecting another vehicle approaching from the front, rear or diagonal-rear direction of one""s own running vehicle using the image picked up and giving a warning to a driver.
2. Description of the Related Art
For example, when a driver of a vehicle running on one-side two lanes of e.g. a speed way intends to change his own vehicle lane, if he changes the lane while he misses another vehicle which catches up with his own vehicle on another lane at a higher speed than his own vehicle from the diagonal-rear direction, there is strong possibility of a serious accident. Therefore, it is desired that the driver surely notices or recognizes other vehicles in the environment.
When another following vehicle on the same lane abruptly approaches his own vehicle from the rear, if the driver of his own vehicle makes abrupt braking, there is possibility of bumping-into-the back.
When another vehicle runs forward on the same lane at a lower speed than that of his own vehicle, if a driver dozes, there is a danger of bumping-into-the back.
Particularly, in recent years, various types of vehicles inclusive of a passenger can have a highly improved performance so that the running speed or acceleration thereof has been greatly increased. Therefore, the vehicle moves very fast in lane changing or acceleration/deceleration. As a result, when the driver misses the presence of another vehicle running in the environment of his own vehicle, there has been increased strong possibility of occurrence of a dangerous state which may lead to serious accidents, or an actual serious accident.
In order to prevent the driver from missing another in the environment than his own vehicle (hereinafter referred to as xe2x80x9csurrounding vehiclexe2x80x9d), the driver himself must first pay attention to the environment. However, it is true that attention or recognition ability of a human being has reached the limit of ability of capable of following the improved performance of a vehicle. Actually, there is an increased tendency of accidents attributable to the missing the surrounding vehicle at the limit value.
Generally, it is known that the visual recognition ability of a human being abruptly decreases as the running speed increases. The vehicle is still desired to deal with a higher running speed in the future. Therefore, there is a serious problem of danger of missing the presence of the surrounding vehicle in e.g. lane changing.
Accordingly, in order to prevent the danger of a great accident or the actual great accident from occurring, only enhancing the attention of the driver to the presence of the surrounding vehicle is not sufficient. It is necessary to recognize the presence of the surrounding vehicle running in the environment automatically, accurately and surely and give the acquired information to the driver so that the limited recognition ability of a human being can be complemented.
Several techniques for avoiding the danger as described above have been proposed as disclosed in JP-A-6-107096 and JP-A-7-50769. Specifically, JP-A-6-107096 discloses a system for detecting an approaching vehicle and collision warning based on the detection of an optical flow. JP-A-7-50769 discloses a warning system which intends to detect an approaching vehicle from the diagonal-rear direction on the basis of detection of an optical flow and issues a warning when there is a danger of colliding with another vehicle in lane changing.
Now referring to FIGS. 20 and 21, an explanation will be given of the summary of a conventional environment monitoring system.
FIGS. 20A-20D are views for explaining a change in a rear background image acquired by a video camera 1. FIG. 20A shows a status inclusive of one""s own vehicle. FIG. 20B shows an image picked up by a video camera 1 at timing t in an environment of one""s own vehicle. FIG. 20C shows an image picked up at timing t+xcex94t.
Now it is assumed that one""s own vehicle is running straight on a flat road. The road sign and building residing in the rear of one""s own vehicle in FIG. 20A are observed as images shown in FIGS. 20B and 20C at timings t and t+xcex94t, respectively. Connecting the corresponding points in these two images provides speed vectors as shown in FIG. 20D. The are referred to as xe2x80x9coptical flowsxe2x80x9d. Where a following vehicle approaches, the directions of the vectors in the optical flows in FIG. 20D are contrary.
These optical flows appear radially from a point called xe2x80x9cFOExe2x80x9d (Focus of Expansion) in the image. The FOE, which refers to xe2x80x9cinfinite pointxe2x80x9d or xe2x80x9cdisappearing pointxe2x80x9d, corresponds to the one point opposite to the running direction of one""s own vehicle on the image where one""s own vehicle runs straight. In this way, the optical flows acquired when one""s own vehicle runs extends radially around the FOE. The optical flows issued from the following vehicle or another vehicle running on an adjacent lane (hereinafter referred to as xe2x80x9cparticular vehiclexe2x80x9d) include information composed of a position and relative speed of the above particular vehicle. Where the optical flows are longer and diverge from the FOE, there is strong possibility of danger.
Now referring to FIG. 21, a detailed explanation will be given of the optical flows. In an optical arrangement shown in FIG. 21, it is assumed that reference numeral 11 denotes a lens of a video camera, 12 denotes an image plane of the video camera, f denotes a distance from the lens 11 to the image plane 12, P (X, Y, Z) denotes any point on the following vehicle, and p (x y) denotes a point corresponding to the point P on the image plane 12.
From the ratio of similarity of a triangle,
x=fxc2x7X/Zxe2x80x83xe2x80x83(1)
By transforming and time-differentiating this equation,
Xxe2x80x2=(xcex94x/xcex94txc2x7Z+Xxc2x7Zxe2x80x2)/fxe2x80x83xe2x80x83(2)
The component u in the x direction of the optical flow is expressed by
u=xcex94x/xcex94txe2x80x83xe2x80x83(3)
Using it, Z can be expressed by
Z=(fxc2x7Xxe2x80x2xe2x88x92xxc2x7Zxe2x80x2)/uxe2x80x83xe2x80x83(4)
Now, Zxe2x80x2 denotes a relative speed between one""s own vehicle and the xe2x80x9cparticular vehiclexe2x80x9d. Assuming that
Zxe2x80x2=xe2x88x92xcex1,xe2x80x83xe2x80x83(5)
Equation (4) can be transformed into
Z=(fxc2x7Xxe2x80x2+xxcex1)/uxe2x80x83xe2x80x83(6)
Therefore, the x direction component of the optical flow (i.e. xcex94x/xcex94t=u) can be expressed by
u=(fxc2x7Xxe2x80x2+xxcex1)/Zxe2x80x83xe2x80x83(7)
This applies to Y.
Thus, from Equation (7), as Z i.e. the distance from the particular vehicle is decreased, otherwise xcex1 is increased (relative speed is large), the x component of the optical flow is increased. This applies to the Y direction.
Therefore, the optical flow is longer as the distance from the xe2x80x9cparticular vehiclexe2x80x9d is shorter or the relative speed is higher. Accordingly, a greater length of the optical flow diverging from FOE provides higher degree of danger for the particular vehicle.
The above processing is repeated for all the points on the image at timing t so that the optical flows of the entire image representative of the danger of each particular vehicle can be acquired. An alarm is sounded according to the acquired degree of danger to alert the driver of his own vehicle to the danger. This complements the limited recognition ability of a human being, and prevents the danger of a great accident or the actual occurrence thereof.
In a prior art, as shown in FIG. 22, in order to save the processing time for an object for which monitoring is not required, the white lines of a lane on which one""s own vehicle runs on a straight road are detected so that the lane on which one""s own vehicle runs is distinguished from an adjacent lane to define a monitoring region. The FOE point is acquired from the extension of the detected white line, and the optical flows extending radially from the FOE point are acquired for one""s own vehicle lane and adjacent lane, thereby detecting a particular vehicle 102 approaching one""s own vehicle. In this way, the particular vehicle is recognized on the basis of the optical flows so that the degree of danger for the particular vehicle can be automatically decided without using any particular distance meter.
However, in the prior art system, on the basis of the optical flows diverging from the FOE which is a reference point, a relative distance and relative speed from the other vehicle is obtained. Therefore, when the FOE is set at a different position from its essential position, the acquired optical flows are also different from the essential ones so that the other vehicle cannot be recognized correctly.
For example, it is assumed that the white lines of a lane on which one""s own vehicle runs on a curved road as shown in FIG. 23 are detected so that the lane on which one""s own vehicle runs is distinguished from an adjacent lane to define a monitoring region. In this case, the other vehicle 100 which is running on one""s own vehicle lane is erroneously detected as if it is running on the adjacent lane to approach one""s own vehicle. Conversely, the other vehicle 103 which is running on the adjacent lane is erroneously detected as if it is running on one""s own vehicle.
Further, where a zebra zone, a pattern or character of speed limitation, etc. are painted on the road surface, optical flows will be produced from these paints. Essentially, it is intended to detect only the optical flow from an approaching vehicle. In this case, however, the optical flows from the paints may be erroneously detected as those from the approaching vehicle.
In view of the problems described above, an object of the present invention is to provide an environment monitoring system capable of automatically and correctly monitoring the approaching of the other forward or following vehicle which runs on one""s own vehicle lane or adjacent lane.
Another object of the present invention is to provide an environment monitoring system and method which can prevent an object (e.g. paint on the road surface) which is not required to prevent from being erroneously detected as another surrounding vehicle.
In order to attain the object of the present invention, there is provided an environment monitoring system for monitoring a relative relation between one""s own vehicle and another surrounding vehicle comprising: image pick-up means for picking up two early and later successive images which are located at a front, rear or diagonal-rear position from one""s own vehicle at two early and later timings and on the basis of the images thus picked-up, detecting white lines of one""s own vehicle lane on which one""s own vehicle runs; monitoring region setting means for setting a monitoring region in one""s own vehicle lane or an adjacent lane on the basis of the white lines thus detected; and optical-flow detecting means for detecting an optical flow generated from another surrounding vehicle within the monitoring region, wherein the monitoring setting means includes: a small region dividing means for dividing the early image into a plurality of small regions, an edge extracting means for extracting an edge which is an candidate of each the white lines for each the small regions using a virtual FOE for each the small regions acquired previously of the early image, a white line determining means for determining each the white lines on the basis of the edge extracted by the edge extracting means, and a virtual FOE estimating means for extending segments of white lines for each the small regions determined by the white line determining means to acquire a coordinate of a crossing point thereof and storing that of the virtual FOE for each the small regions whereby the virtual FOE is used to extract the edge for each the small regions of the later image.
In this configuration, since the shifting amount in the virtual FOE between the successive images apart by a short timing is minute, the edge which constitutes a candidate of a white line for each of the corresponding small regions of a newly picked-up image can be extracted, thereby surely determining the white lines.
Preferably, the edge extracting means comprises: means for gradually shifting a fan-like region by a predetermined angle around the coordinate of the virtual FOE to extract the edge for each fan-like region, and means for taking the extracted edges whose number exceeds a threshold value as candidates of each the white lines.
In this configuration, the edge having a greatly different direction from that of an actual white line can be removed as noise.
Preferably, the white line determining means performs linear approximation for the edges to acquire the corresponding line segments and determines the line segments satisfying continuity between the small regions as the white lines. In this configuration, the linear segment which is not successive to the linear segment of the small region previously known by linear approximation can be excluded from a candidate of the white line.
Preferably, the white line determining means comprises: means for acquiring the coordinates and angles of crossing points formed by extending all pairs of white line segments in each of the small regions subjected to the linear approximation; and means for acquiring differences between the coordinates and angles thus acquired and the coordinate of the virtual FOE and angle of two white line segments in each the small regions which have been acquired previously from the early image so that a pair of lines providing minimum values in coordinate and angle are determined as the white lines of the small region at issue.
In this configuration, two left and right linear segments which appear to be white lines can be selected at a time from a plurality of linear segments.
Preferably, the optical-flow detecting means comprises: means for reverse-projection converting the early image on the basis of a predetermined optical arrangement of the image pick-up means onto an x-z plane in parallel to a road surface in a real space to acquire a road surface image; means for computing a moving distance of one""s own vehicle between the two timings on the basis of a time interval between the two timings and speed information of one""s own vehicle; means for parallel-shifting the road surface image by the moving distance thus computed; means for computing angles of the white lines in the vicinity of one""s own vehicle formed together with a z-axis on the road surface image parallel-shifted to acquire an average angle thereof; means for rotating the road surface by the average angle around an origin of the x-z coordinate; means for projection-converting the road surface image after parallel-shifted to acquire an estimated image of the later image in timing; means for acquiring a differential image between the later image and its estimated image to extract a feature point; and means for searching a corresponding point of the feature point extracted, thereby detecting the optical flow.
In this configuration, since an undesired object such as the shadow, character, dirt, etc. on the road surface whose optical flow can be canceled, with only the object, whose optical flow should be detected, being left, searching of the corresponding points can be performed.
Preferably, the optical flow detecting means searches the corresponding point of the feature point in each the small regions in only a radial direction of the FOE. In this configuration, the number of the feature points for which the corresponding points should be searched in each of the small regions can be reduced greatly.
Preferably, the environment monitoring system further comprises means for evaluating the degree of danger which decides dangerous when the average value of the magnitudes of the optical flows 1 as regards the degree D of danger in a front, rear or diagonal-rear field of view expressed by   D  =      l    =                  (                  1          /          N                )            xc3x97                        ∑                      i            =            1                    N                ⁢                  xe2x80x83                ⁢        li            
(where li denotes the magnitude of the optical flow, and N denotes the number of generated optical flows) exceeds a prescribed value.
In this configuration, the degree of danger can be evaluated irrespectively of the number of optical flows generated which depends on the type, color and size of a surrounding vehicle, and the brightness of the environment (daytime and nighttime).
Preferably, the means for evaluating the degree of danger decides dangerous when the number N of the optical flows generated exceeds a prescribed value.
In this configuration, even the distance between one""s own vehicle and forward vehicle is shorter, the degree of danger can be surely decided.
Preferably, means for evaluating the degree of danger evaluates the degree of danger for each the small regions.
In accordance with another aspect of the present invention, there is provided a method for monitoring a relative relation between one""s own vehicle and another surrounding vehicle comprising the steps of: picking up two early and later successive images which are located at a front, rear or diagonal-rear position from one""s own vehicle at two early and later timings and on the basis of the images thus picked-up, detecting white lines of one""s own vehicle lane on which one""s own vehicle runs; setting a monitoring region in one""s own vehicle lane or an adjacent lane on the basis of the white lines thus detected; and detecting an optical flow generated from another surrounding vehicle within the monitoring region, wherein the monitoring region setting step includes the sub-steps: dividing the early image into a plurality of small regions, extracting an edge which is an candidate of each the white lines for each the small regions using a virtual FOE for each the small regions acquired previously of the early image, determining each the white lines on the basis of the edge extracted by the edge extracting means, and extending segments of white lines for each small region determined by the white line determining means to acquire a coordinate of a crossing point thereof and storing that of the virtual FOE for each the small regions whereby the virtual FOE is used to extract the edge for each the small regions.
In this configuration, since the shifting amount in the virtual FOE between the successive images apart by a short timing is minute, the edge which constitutes a candidate of a white line for each of the corresponding small regions of a newly picked-up image can be extracted, thereby easily detecting the curved state of a road for the small region.